UN CO RR EC TE D PR OO F 2 Assessment of dialogue systems by means of a new 3 simulation technique

7 Abstract 8 In recent years, a question of great interest has been the development of tools and techniques to facilitate the 9 evaluation of dialogue systems. The latter can be evaluated from various points of view, such as recognition and un-10 derstanding rates, dialogue naturalness and robustness against recognition errors. Evaluation usually requires com-11 piling a large corpus of words and sentences uttered by users, relevant to the application domain the system is designed 12 for. This paper proposes a new technique that makes it possible to reuse such a corpus for the evaluation and to check 13 the performance of the system when different dialogue strategies are used. The technique is based on the automatic 14 generation of conversations between the dialogue system, together with an additional dialogue system called user 15 simulator that represents the user's interaction with the dialogue system. The technique has been applied to evaluate a 16 dialogue system developed in our lab using two different recognition front-ends and two different dialogue strategies to 17 handle user confirmations. The experiments show that the prompt-dependent recognition front-end achieves better 18 results, but that this front-end is appropriate only if users limit their utterances to those related to the current system 19 prompt. The prompt-independent front-end achieves inferior results, but enables front-end users to utter any permitted 20 utterance at any time, irrespective of the system prompt. In consequence, this front-end may allow a more natural and 21 comfortable interaction. The experiments also show that the re-prompting confirmation strategy enhances system 22 performance for both recognition front-ends. 25 1. Introduction 26 Dialogue systems have developed greatly in the last decade, such systems being applied to airline and 27 railway information, weather forecasts, ticket booking, automatic telephone switchboards, multimedia 28 services, routing systems, etc.